Skip to content

Implement subresources in VWs - #4342

Open
ntnn wants to merge 14 commits into
kcp-dev:mainfrom
ntnn:vw-subresources-pr
Open

Implement subresources in VWs#4342
ntnn wants to merge 14 commits into
kcp-dev:mainfrom
ntnn:vw-subresources-pr

Conversation

@ntnn

@ntnn ntnn commented Aug 26, 2026

Copy link
Copy Markdown
Member

Summary

Implements subresources in VWs, specifically the TokenRequest.
Went with RBAC style in the claims for less API changes and because it feels closer. I tested a separate Subresoure as part of the claims but that didn't look quite right.

I've been hacking on this on and off for two weeks.
I think its secure now because to access a subresource a) the claim for the subresource needs to be present and b) the parent resource needs to be claimed as well, so the usual gates apply.
subresource claims cannot have their own labels to prevent mismatching labels between parent and subresource (I could see the value of being able to distinguish that a provider has access to resource set A but only to subset B for accessing the subresource - but that feels like the edge case of an edge case).

What Type of PR Is This?

/kind feature

Related Issue(s)

Fixes #

Release Notes

VWs can now handle subresources other than status of core APIs such as TokenRequest for ServiceAccount

ntnn added 5 commits August 26, 2026 23:08
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
@kcp-ci-bot kcp-ci-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has signed the DCO. kind/feature Categorizes issue or PR as related to a new feature. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API labels Aug 26, 2026
@kcp-ci-bot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign embik for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kcp-ci-bot kcp-ci-bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Aug 26, 2026
@ntnn
ntnn force-pushed the vw-subresources-pr branch from 60af7fe to 3fc8c0e Compare August 26, 2026 22:42
ntnn added 6 commits August 27, 2026 08:02
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
127 because a resource can have up to 63 bytes per segment, so 63 for
the resource, 63 for the subresource and 1 for the `/`.

Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
@ntnn
ntnn force-pushed the vw-subresources-pr branch from 3fc8c0e to 4aba907 Compare August 27, 2026 06:02
@ntnn

ntnn commented Aug 27, 2026

Copy link
Copy Markdown
Member Author

/retest

Signed-off-by: Nelo-T. Wallus <n.wallus@sap.com>
@ntnn

ntnn commented Aug 27, 2026

Copy link
Copy Markdown
Member Author

/retest

One is #4343
The other === FAIL: test/e2e/apibinding TestAPIBindingPermissionClaimsAppliedAcrossShards (43.62s) looks like a flake

@ntnn

ntnn commented Aug 27, 2026

Copy link
Copy Markdown
Member Author

/retest

infra failure

@ntnn

ntnn commented Aug 27, 2026

Copy link
Copy Markdown
Member Author

/retest

 + set +o xtrace
Command terminated with 0
Compressing build artifacts...
tar: kcp/audit.log: file changed as we read it
tar: kcp: Cannot rmdir: Directory not empty
tar: Exiting with failure status due to previous errors

@gman0

gman0 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Nice!

/lgtm

I'll leave approval for a bit if someone else wants to have a look.

@kcp-ci-bot kcp-ci-bot added the lgtm Indicates that a PR is ready to be merged. label Aug 27, 2026
@kcp-ci-bot

Copy link
Copy Markdown
Contributor

LGTM label has been added.

DetailsGit tree hash: 0bb5af6988e649277b698d6e26a194d0f83dfff4

Comment on lines +134 to +135
// subresource status is implicitly granted with its parent resource.
// others must be claimed explicitly

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this always the case?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we just never checked for subresource permissions because only implemented status so far.
Given kube also doesn't give implicit access to the status subresource - but requiring this now would break existing APIExport/-Bindings.
I wanted to bring this up in todays community meeting to see how we feel about this.
On one hand I'd rather we are exact, on the other I'm not sure how we could make this "nice" for end users.

E.g. we could gate the implicit status on a feature gate, but that isn't really nice.
We could just keep status being implicit for now and require it explicitly in the next API version.
But I'm not really a fan of either.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Albeit not on paper, but the "spirit" of offering a service is for the consumer to give the spec, and provider to update the status - so I think it's assumed the provider needs access to that at all times, even when claiming. So if we're looking for an excuse to keep these perms implicit, I think this could be one :D but it does sound a bit weak.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean yeah but for resources exported via an APIExport we are already implicitly claiming the resource and the subresources^^
But I get what you mean. Plus that subresources are only really interesting for update/patch, so at that point the consumer is already accepting mutations from the provider on the claimed resource - and that could include subresources.

@@ -119,6 +124,50 @@ func provideDelegatingRestStorage(ctx context.Context, dynamicClusterClientFunc

// TODO(sttts): add scale subresource

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this TODO still relevant?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. CRDs can support the scale subresource, so technically a CRD could be exported via an APIExport that supports the scale subresource.
https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#scale-subresource

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But why is it different than status? Isn't this PR working for all subresources?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not all, but making it possible to handle subresources other than status with the changes to permission claims and handling it in authorization.

I can take a stab at implementing the scale subresource as well for completeness :D
I'm not sure if scale is actually different or if sttts just didn't bother with it at the time.

But I think there's more work down the line because I'm not sure how well this works for e.g. virtual resource. But I haven't used virtual resources at all so far so 🤷

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kk

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done :D Scale subresource works!

ntnn added 2 commits August 28, 2026 18:51
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
Signed-off-by: Nelo-T. Wallus <red.brush9525@fastmail.com>
@kcp-ci-bot kcp-ci-bot removed the lgtm Indicates that a PR is ready to be merged. label Aug 28, 2026
@kcp-ci-bot
kcp-ci-bot requested a review from gman0 August 28, 2026 16:51
@kcp-ci-bot

Copy link
Copy Markdown
Contributor

New changes are detected. LGTM label has been removed.

@kcp-ci-bot kcp-ci-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 28, 2026

@mjudeikis mjudeikis left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few other comments:

Two silent no-ops slip through admission:

an export claiming only serviceaccounts/token without claiming serviceaccounts → no API served, no error, no condition;
a CRD claim for a nonexistent subresource (cowboys/notreal + identityHash) → accepted, dead.

I have more things, but I think all and all we need better e2e:

  1. Non existing resource
  2. Only status claim
  3. Inherited status permissions (status does not have claims)
    ....

- verbs
type: object
x-kubernetes-validations:
- message: subresource claims must use selector.matchAll

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this meant that if I claim named resource virtualmachine I will get all other virtualmachines subresources?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It just means that you can apply a selector on top of the selectors for the parent resource.

e.g. you have the virtualmachines resource with label my.exposed.resources=true and then you claim the subresource virtualmachines/ssh you cannot apply another label filter on top.
I had that implemented and I see some use for it (I mentioned that in the description) but I don't think that too many people will use it.
The story is basically "when you claim a subresource you must have access to its parent".
Hence the CEL validation that the selector must be matchall.

ab.SetLabels(lbls)
}

// subresource claims have no selector, set a matchAll so users

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sets MatchAll = trueunconditionally, even when the user explicitly set matchLabels/matchExpressions on the subresource claim. ValidateAPIBindingPermissionClaims then rejects with "matchLabels cannot be used with matchAll" — for a matchAll the user never wrote, which will be very confusing to debug :)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The user cannot set matchlabels/-expression; I added a CEL rule for that.
Maybe the rule doesn't hold in which case I have to fix that :D

// not provided by an api export.
//
// +kubebuilder:validation:Pattern=`^[a-z][-a-z0-9]*[a-z0-9]$`
// +kubebuilder:validation:Pattern=`^[a-z][-a-z0-9]*[a-z0-9](/[a-z][-a-z0-9]*[a-z0-9])?$`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bit of AI explaining as it did better job than I did. I think this is important one , if not most. I tried to look to this via "controller-runtime generated patterns"

What the API now allows:
The new regex on resource accepts anything shaped like foo/bar — including cowboys/status. Nothing anywhere rejects .../status as a claim. So you can write it in an APIExport and APIBinding, and it gets accepted and stored.

What the authorizer actually does with status requests:
When a request comes in for a status subresource, the authorizer deliberately ignores the subresource and looks for a claim on the parent resource only:

// binding.go
claimedResource := normalizedGR.Resource            // "cowboys"
if sub != "" && sub != "status" {                   // status is skipped!
    claimedResource = resource + "/" + sub
}

So for a status update, it searches your claims for cowboys — never for cowboys/status.

The trap:
Imagine a provider who thinks in RBAC terms (which the docs literally encourage — "in the style of RBAC rules"). In RBAC you'd write:

cowboys → verbs get, list (read the objects)
cowboys/status → verbs update (write only the status)

That's a completely reasonable, even best-practice layout. But here's what happens:

  1. Both claims are accepted. No error, no warning. ✅ (looks fine)
  2. Provider tries to update a cowboy's status.
  3. Authorizer looks for a claim on cowboys with verb update.
  4. The cowboys claim only has get, list → denied.
  5. The cowboys/status claim with update? Never consulted. It's a dead claim — it exists, it's accepted, and it does absolutely nothing.
  6. The user gets a permission denied and has zero indication that their cowboys/status claim is being ignored. They'll stare at it and think "but I claimed it right there!"

Why "accepting-but-ignoring is the worst of both":
There are two consistent designs:

Status is implicit → then writing foo/status should be rejected with a clear message ("you don't need this, status comes with the parent claim").
Status is explicit → then foo/status claims should actually work in the authorizer.

The PR does neither: it accepts the claim syntactically but ignores it semantically. That's the worst combination, because the API lies to the user.

The suggested fix (cheap one):
Add a CEL validation so it's impossible to write:

rule: "!self.resource.endsWith('/status')"
message: "status is granted implicitly by the parent resource claim"

Now the user who writes cowboys/status gets an immediate, self-explanatory error at creation time instead of a silent no-op at request time.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and we need e2e to for this

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix the AI suggested with not allowing the status subresource is what I mentioned in call earlier; but before we make a decision on that I think we should discuss it in the cc to see what people think.

Personally I would prefer having status implicit for this API version and not allow setting it and requiring it to be explicit in the next API version.

// ScopedPermissionClaim embeds a PermissionClaim and adds a selector to
// scope down access to objects of the claimed resource.
//
// +kubebuilder:validation:XValidation:rule="!self.resource.contains('/') || (has(self.selector.matchAll) && self.selector.matchAll)",message="subresource claims must use selector.matchAll"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is missleading a bit.

Maybe: "for subresource claims the selector is ignored; scope is inherited from the parent resource claim".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API kind/feature Categorizes issue or PR as related to a new feature. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants